In many contexts, simpler models are preferable to more complex models and the control of this model complexity is the goal for many methods in machine learning such as regularization, hyperparameter tuning and architecture design. In deep learning, it has been difficult to understand the underlying mechanisms of complexity control, since many traditional measures are not naturally suitable for deep neural networks. Here we develop the notion of geometric complexity, which is a measure of the variability of the model function, computed using a discrete Dirichlet energy. Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, flatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.

Interestingly, most of these works are either not applicable to, or have not been tested on, node-level tasks.

However,theperformance ofexisting GNNs would decrease significantly when they stack many layers, because of the oversmoothing issue. Node embeddings tend to converge to similar vectors when GNNs keep recursively aggregating the representations ofneighbors.

Gradient-Guided Dynamic Rewiring of GCNs.Contrary toad-hoc addition of skipconnections toimproveGCNs performance, inthis paper,we leverage Gradient Flowto introduce dynamic rewiring strategyof vanilla-GCNs with skip-connections.